Depth sensor based passenger sensing for passenger conveyance control

ABSTRACT

A passenger conveyance system includes a depth-sensing sensor for capturing depth map data of objects within a field of view adjacent a passenger conveyance door. A processing module in communication with the depth-sensing sensor to receive the depth map data, the processing module uses the depth map data to track an object and calculate passenger data associated with the tracked object, and a passenger conveyance controller to receive the passenger data from the processing module, wherein the passenger conveyance controller controls a passenger conveyance dispatch control function in response to the passenger data.

BACKGROUND

The present disclosure relates to a passenger conveyance and, moreparticularly, to a depth sensor based control for an elevator.

Elevator performance can be derived from a number of factors. To anelevator passenger, an important factor can include travel time. Forexample, as time-based parameters are minimized, passenger satisfactionwith the service of the elevator can improve. Modern elevator systemsmay still provide opportunities for improved passenger experience andtraffic performance.

SUMMARY

A passenger conveyance system according to one disclosed non-limitingembodiment of the present disclosure can include a depth-sensing sensorfor capturing depth map data of objects within a field of view adjacenta passenger conveyance door; a processing module in operablecommunication with the depth-sensing sensor to receive the depth mapdata, the processing module using the depth map data to track an objectand calculate passenger data associated with the tracked object; and apassenger conveyance controller to receive the passenger data from theprocessing module, wherein the passenger conveyance controller controlsa passenger conveyance dispatch control function in response to thepassenger data.

A further embodiment of the present disclosure may include, wherein thedepth map data is 3D depth map data.

A further embodiment of any of the foregoing embodiments of the presentdisclosure may include, wherein the depth-sensing sensor or technologycomprises a structured light measurement, phase shift measurement, timeof flight measurement, stereo triangulation device, sheet of lighttriangulation device, light field cameras, coded aperture cameras,computational imaging techniques, simultaneous localization and mapping(SLAM), imaging radar, imaging sonar, scanning LIDAR, flash LIDAR,Passive Infrared (PIR) sensor, and small Focal Plane Array (FPA), or acombination comprising at least one of the foregoing.

A further embodiment of any of the foregoing embodiments of the presentdisclosure may include, wherein the field-of-view is about 180 degrees.

A further embodiment of any of the foregoing embodiments of the presentdisclosure may include, wherein the processing module calculates anobject parameter of the tracked object, wherein the object parametercomprises an object count, a location, a size, a direction, anacceleration, a velocity, an object classification, or a combinationcomprising at least one of the foregoing.

A further embodiment of any of the foregoing embodiments of the presentdisclosure may include, wherein the processing module is incommunication with the passenger conveyance controller and communicatesthe object parameter to the passenger conveyance controller.

A further embodiment of any of the foregoing embodiments of the presentdisclosure may include, wherein the processing module calculates thepassenger data based on the object parameter, wherein the passenger dataprovided to the passenger conveyance controller comprises an estimatedarrival time, a probability of arrival, a covariance, a number ofpassengers waiting for a passenger conveyance, or a combinationcomprising at least one of the foregoing.

A further embodiment of any of the foregoing embodiments of the presentdisclosure may include, wherein the object parameter of the trackedobject comprises object classification and the processing modulecalculates the passenger data if the object classification comprises apassenger.

A further embodiment of any of the foregoing embodiments of the presentdisclosure may include, wherein the processing module divides a field ofview of the depth sensing sensor into a first region and a secondregion, wherein the second region is defined as an area adjacent thepassenger conveyance door.

A further embodiment of any of the foregoing embodiments of the presentdisclosure may include, wherein the passenger data comprises the numberof passengers waiting for a passenger conveyance, and wherein theprocessing module increments the number of passengers waiting for apassenger conveyance based on a number of tracked objects that enter thesecond region.

A further embodiment of any of the foregoing embodiments of the presentdisclosure may include, wherein the depth-sensing sensor is located on awall adjacent to a passenger conveyance door.

A further embodiment of any of the foregoing embodiments of the presentdisclosure may include, wherein the depth-sensing sensor is located at150 mm to 700 mm from the adjacent floor.

A further embodiment of any of the foregoing embodiments of the presentdisclosure may include, wherein the depth-sensing sensor is located atabout knee height.

A further embodiment of any of the foregoing embodiments of the presentdisclosure may include, wherein the field of view comprises a passengerwaiting area adjacent to the passenger conveyance door.

A further embodiment of any of the foregoing embodiments of the presentdisclosure may include, wherein the field of view comprises thepassenger conveyance door and an area at least partially surrounding thepassenger conveyance door.

A method of providing depth map data for use in passenger conveyancecontrol, the method according to another disclosed non-limitingembodiment of the present disclosure can include detecting an objectlocated in a field of view of a depth-sensing sensor; tracking theobject based on distance to the passenger conveyance door; calculatingpassenger data associated with the tracked object; communicating thepassenger data to a passenger conveyance controller; and controlling apassenger conveyance cab with the passenger conveyance controller inresponse to the passenger data.

A further embodiment of any of the foregoing embodiments of the presentdisclosure may include, wherein controlling the passenger conveyance cabfurther comprises opening the passenger conveyance door, moving thepassenger conveyance cab, stopping the passenger conveyance cab,redirecting the passenger conveyance cab, or a combination comprising atleast one of the foregoing.

A further embodiment of any of the foregoing embodiments of the presentdisclosure may include, wherein controlling further comprisesdispatching two or more passenger conveyance cabs substantiallysimultaneously in response to the passenger data.

A further embodiment of any of the foregoing embodiments of the presentdisclosure may include, wherein calculating passenger data includescalculating an object parameter for the tracked object, wherein theobject parameter includes location, a size, a velocity, a direction, anacceleration, an object classification, or a combination comprising atleast one of the foregoing.

A further embodiment of any of the foregoing embodiments of the presentdisclosure may include, wherein calculating passenger data comprisesbackground subtraction.

A further embodiment of any of the foregoing embodiments of the presentdisclosure may include, wherein calculating passenger data comprisesframe differencing.

A further embodiment of any of the foregoing embodiments of the presentdisclosure may include, wherein calculating passenger data comprisesspurious data rejection.

A further embodiment of any of the foregoing embodiments of the presentdisclosure may include, wherein spurious data rejection includes:computing a depth background; segmenting a foreground object; removing aforeground region; segmenting a moving object by a 3D morphologicaloperation; transforming the moving object to 3D world coordinates;estimating an actual height and actual volume of a moving object; andremoving a spurious moving object from a scene by geometric filtering.

A further embodiment of any of the foregoing embodiments of the presentdisclosure may include, wherein the 3D morphological operation comprisescomputing a 2D foreground object by depth background subtraction, sizefiltering on a mask as a function of range; connecting mask regions;segmenting objects in 3D based on depth discontinuity, or a combinationcomprising at least one of the foregoing.

A further embodiment of any of the foregoing embodiments of the presentdisclosure may include, wherein the 2D foreground object within the maskis at any depth.

The foregoing features and elements may be combined in variouscombinations without exclusivity, unless expressly indicated otherwise.These features and elements as well as the operation thereof will becomemore apparent in light of the following description and the accompanyingdrawings. It should be appreciated, however, the following descriptionand drawings are intended to be exemplary in nature and non-limiting.

BRIEF DESCRIPTION OF THE DRAWINGS

Various features will become apparent to those skilled in the art fromthe following detailed description of the disclosed non-limitingembodiment. The drawings that accompany the detailed description can bebriefly described as follows:

FIG. 1 is a schematic view of an elevator system according to onedisclosed non-limiting embodiment;

FIG. 2 is a block diagram of an elevator system according to anotherdisclosed non-limiting embodiment;

FIG. 3 is a perspective view of an elevator system according to anotherdisclosed non-limiting embodiment;

FIG. 4 is a block diagram of an algorithm for an elevator systemaccording to another disclosed non-limiting embodiment;

FIG. 5 is a block diagram of an algorithm for an elevator systemaccording to another disclosed non-limiting embodiment;

FIG. 6 is a block diagram for an elevator system according to anotherdisclosed non-limiting embodiment;

FIG. 7 is a block diagram of an algorithm for an elevator systemaccording to another disclosed non-limiting embodiment;

FIG. 8 is a block diagram for an elevator system according to anotherdisclosed non-limiting embodiment;

FIG. 9 is a block diagram of an algorithm for an elevator systemaccording to another disclosed non-limiting embodiment;

FIG. 10 is a block diagram of an algorithm for an elevator systemaccording to another disclosed non-limiting embodiment;

FIG. 11 is a block diagram of an algorithm for an elevator systemaccording to another disclosed non-limiting embodiment;

FIG. 12 is a block diagram for an elevator system according to anotherdisclosed non-limiting embodiment;

FIG. 13 is a block diagram of an algorithm for an elevator systemaccording to another disclosed non-limiting embodiment;

FIG. 14 is a schematic view illustrating operation of an elevator systemaccording to another disclosed non-limiting embodiment;

FIG. 15 is a block diagram for an elevator system according to anotherdisclosed non-limiting embodiment;

FIG. 16 is a block diagram of an algorithm for an elevator systemaccording to another disclosed non-limiting embodiment;

FIG. 17 is a schematic view a human tracker for an elevator systemaccording to another disclosed non-limiting embodiment;

FIG. 18 is a graphical representation of statistical heights for anelevator system according to another disclosed non-limiting embodiment;

FIG. 19 is a block diagram for an elevator system according to anotherdisclosed non-limiting embodiment;

FIG. 20 is a block diagram of a table for an elevator system accordingto another disclosed non-limiting embodiment;

FIG. 21 is a block diagram of an algorithm for an elevator systemaccording to another disclosed non-limiting embodiment;

FIG. 22 is a graphical representation for passenger tracking from anorigin passenger waiting area to a destination passenger waiting areavia in-car tracking;

FIG. 23 is a schematic view of a door arrangement for an elevator systemaccording to another disclosed non-limiting embodiment;

FIG. 24 is a block diagram of an elevator system according to anotherdisclosed non-limiting embodiment; and

FIG. 25 is a schematic view of traffic list generation for a singleuser; and

FIG. 26 is a block diagram of an algorithm for an elevator system.

DETAILED DESCRIPTION

FIG. 1 schematically illustrates a passenger conveyance system 20 suchas an elevator system. The system 20 can include an elevator car 22, anelevator door 24, a lobby call 26, a car-operating panel (COP) 28, asensor system 30, and a control system 32. It should be appreciated thatalthough an elevator system is disclosed and illustrated as an exampleherein, other passenger conveyance systems such as mass transitvehicles, access control passenger conveyance through various securecheckpoints, triggering video monitoring, hotel room access, and otherdetection, security, and identification, will also benefit herefrom.That is, passenger conveyance may be broadly construed as controlsassociated with passage of an individual. It should be furtherappreciated that although particular systems are separately defined,each or any of the systems may be otherwise combined or separated viahardware and/or software.

The overall amount of travel time a passenger associates with elevatorperformance may include three time intervals. A first time interval canbe the amount of time a passenger waits in a lobby for an elevator toarrive, hereafter the “wait time.” A second time interval can be the“door dwell time” or the amount of time the elevator doors are open,allowing passengers to enter or leave the elevator. A third timeinterval can be the “ride time” or amount of time a passenger spends inthe elevator. The ride time can also include a stop on an intermediatefloor to allow passengers to enter and/or exit the elevator which canadd to the ride time by at least the door dwell time during the stop.

Various elevator systems can utilize a passenger-initiated input tosignal the need for service. For example, input from the lobby call 26may include a push button, e.g., up, down, or desired destination, torequest elevator service. The passenger initiated input (e.g., via acall button) may notify the control system 32 of the presence of apassenger awaiting elevator service. In response, the control system 32may dispatch the elevator car 22 to the appropriate floor. Optionally,once inside the elevator car 22, the passenger may push a button on thecar-operating panel (COP) 28 designating the desired destination,direction, or the like, and then the control system 32 may dispatch theelevator car 22 to that destination.

The control system 32 can include a control module 40 with a processor42, a memory 44, and an interface 46. The control module 40 can includea portion of a central control, a stand-alone unit, or other system suchas a cloud-based system. The processor 42 can include any type ofmicroprocessor having desired performance characteristics. The memory 44may include any type of computer readable medium that stores the dataand control processes disclosed herein. That is, the memory 44 is anexample computer storage media that can have embodied thereoncomputer-useable instructions such as a process that, when executed, canperform a desired method. The interface 46 of the control module 40 canfacilitate communication between the control module 40 and othersystems.

With reference to FIG. 2, a depth-sensor based passenger sensing system60 can include a sensor 62 that communicates with a data capture module64, and a processing module 66. The depth-sensor based passenger sensingsystem 60 can be a portion of the control system 32, a stand-alone unit,or other system such as a cloud-based system in communication with thecontrol system 32. The data capture module 64, and the processing module66 can be particular to the sensor 62 to acquire and process the datatherefrom. In one example, the sensor 62, through the data capturemodule 64 and the processing module 66, is operable to obtain depth mapdata such as the presence of a passenger in a passenger waiting area orlobby H, an estimated time of arrival (ETA) of the passenger, a numberof passengers in the lobby H, etc.

The sensor 62, according to one disclosed non-limiting embodiment, canbe installed in a lower portion of wall W of the lobby H such as at kneeheight (FIG. 3). The sensor 62 in this disclosed non-limiting embodimentincludes a depth-sensing sensor. It should be appreciated that the term“sensor,” is used throughout this disclosure for any 1D, 2D, or 3D depthsensor, or combination thereof. Such a sensor can be operable in theoptical, electromagnetic or acoustic spectrum capable of producing adepth map (also known as a point cloud or occupancy grid) of thecorresponding dimension(s). Various depth sensing sensor technologiesand devices include, but are not limited to, a structured lightmeasurement, phase shift measurement, time of flight measurement, stereotriangulation device, sheet of light triangulation device, light fieldcameras, coded aperture cameras, computational imaging techniques,simultaneous localization and mapping (SLAM), imaging radar, imagingsonar, scanning LIDAR, flash LIDAR, Passive Infrared (PR) sensor, andsmall Focal Plane Array (FPA), or a combination comprising at least oneof the foregoing. Different technologies can include active(transmitting and receiving a signal) or passive (only receiving asignal) and may operate in a band of the electromagnetic or acousticspectrum such as visual, infrared, etc. The use of depth sensing canhave specific advantages over conventional 2D imaging. The use ofinfrared sensing can have specific benefits over visible spectrumimaging such that alternatively, or additionally, the sensor can be aninfrared sensor with one or more pixels of spatial resolution, e.g., aPassive Infrared (PIR) sensor or small IR Focal Plane Array (FPA).

Notably, there can be qualitative and quantitative differences between2D imaging sensors, e.g., conventional security cameras, and 1D, 2D, or3D depth sensing sensors to the extent that the depth-sensing providesnumerous advantages. In 2D imaging, the reflected color (mixture ofwavelengths) from the first object in each radial direction from theimager is captured. The 2D image, then, can include the combinedspectrum of the source illumination and the spectral reflectivity ofobjects in the scene. A 2D image can be interpreted by a person as apicture. In 1D, 2D, or 3D depth-sensing sensors, there is no color(spectral) information; rather, the distance (depth, range) to the firstreflective object in a radial direction (1D) or directions (2D, 3D) fromthe sensor is captured. 1D, 2D, and 3D technologies may have inherentmaximum detectable range limits and can be of relatively lower spatialresolution than typical 2D imagers. The use of 1D, 2D, or 3D depthsensing can advantageously provide improved operations compared toconventional 2D imaging in their relative immunity to ambient lightingproblems, better separation of occluding objects, and better privacyprotection. The use of infrared sensing can have specific benefits overvisible spectrum imaging. For example, a 2D image may not be able to beconverted into a depth map nor may a depth map have the ability to beconverted into a 2D image (e.g., an artificial assignment of contiguouscolors or grayscale to contiguous depths may allow a person to crudelyinterpret a depth map somewhat akin to how a person sees a 2D image, itis not an image in the conventional sense.). This inability to convert adepth map into an image might seem a deficiency, but it can beadvantageous in certain analytics applications disclosed herein.

The sensor 62 can be, in one example, an eye-safe line-scan LIDAR inwhich the field-of-view (FOV) can be, for example, about 180 degrees,which can horizontally cover the entire area of a lobby or otherpassenger area adjacent to the elevator doors 24 (FIG. 2). The output ofthe LIDAR may, for example, be a 2D horizontal scan of the surroundingenvironment at a height where the sensor 62 is installed. For an activesensor, each data point in the scan represents the reflection of aphysical object point in the FOV, from which range and horizontal angleto that object point can be obtained. The scanning rate of LIDAR can be,for example, 50 ms per scan, which can facilitate a reliable track of apassenger. That is, before application of analytic processes via theprocessing module 66, the LIDAR scan data can be converted to anoccupancy grid representation. Each grid represents a small region,e.g., 5 cm×5 cm. The status of the grid can be indicated digitally,e.g., 1 or 0, to indicate whether each grid square is occupied. Thus,each data scan can be converted to a binary map and these maps then usedto learn a background model of the lobby, e.g. by using processesdesigned or modified for depth data such as a Gaussian Mixture Model(GMM) process, principal component analysis (PCA) process, a codebookprocess, or a combination including at least one of the foregoing.

The processing module 66 may utilize various 3D detection and trackingprocesses (disclosed elsewhere herein) such as background subtraction,frame differencing, and/or spurious data rejection that can make thesystem more resistant to spurious data. Such spurious data can beinherent to depth sensing and may vary with the particular technologyemployed. For active techniques, where a particular signal is emittedand subsequently detected to determine depth (e.g., structured light,time of flight, LIDAR, and the like) highly reflective surfaces mayproduce spurious depth data, e.g., not the depth of the reflectivesurface itself, but of a diffuse reflective surface at a depth that isthe depth to the reflective surface plus the depth from the reflectivesurface to some diffusely reflective surface. Highly diffuse surfacesmay not reflect a sufficient amount of the transmitted signal todetermine depth that may result in spurious gaps in the depth map. Evenfurther, variations in ambient lighting, interference with other activedepth sensors or inaccuracies in the signal processing may result inspurious data.

With reference to FIGS. 4 and 5, in another disclosed non-limitingembodiment, processes 50, 51 for rejection of spurious data aredisclosed in terms of functional block diagrams. These functions can beenacted in dedicated hardware circuitry, programmed software routinescapable of execution in a microprocessor based electronic controlsystem, or a combination including at least one of the foregoing.

Spurious data rejection process 50 can include multiple steps. First, adepth background can be computed which can be used to segment foregroundobjects, e.g., a passenger, luggage, etc., from the background, e.g.,walls and floors (step 52). The depth data may be three-dimensional. Itshould be appreciated that the depth data may alternatively be referredto as a depth map, point cloud, or occupancy grid. The depth data may berelatively “noisy.”

A multi-dimensional-based approach can be used to model the depthbackground. 2D imager background modeling approaches can be insufficientfor depth background modeling. For example, the depth uncertainty can bean analytical function of range, the depth data error can bediscontinuous (or not continuous), and the depth distribution can benon-Gaussian as compared to typical 2D image data (e.g., not able to berepresented by a continuous probability distribution), or a combinationcomprising at least one of the foregoing which can render the 2D imagerbackground modeling insufficient for depth background modeling.

Second, after background subtraction and foreground detection,morphological operations may be used to filter isolated small foregroundregions (e.g., which can be “noise”) and to segment moving objects,called blobs, for further analysis (step 54). This analysis can beperformed in 3D. However, 3D extension of 2D connected components may beinappropriate since the 3D data still has self-occlusion, e.g.,“shadows” in an occupancy grid. An approach to filtering may include aprocess of extending 2D connected components to include an “unknown”category in the occupancy grid for 3D morphological filtering.

The morphological filtering is further explained in reference to FIG. 5.In 3D morphological filtering 51, accounting for occlusion can beperformed in multiple steps (e.g., as shown in FIG. 5 which can includefour successive steps). A 2D foreground object mask can be computed bydepth background subtraction (step 53). Foreground objects within themask can be at any depth, and partially or completely occlude objectstherebehind.

Size filtering can be performed on the mask as a function of range thatmay remove objects below a predetermined size (step 55). Any “nearby”mask regions are connected using 2D connected components thatpotentially merge objects with distinct depths (step 57). The objectscan then be segmented in 3D based on depth discontinuity (step 59). Itis possible that some objects after depth discontinuity segmentationwill be relatively small, e.g., someone almost entirely occluded byanother person will appear as a small blob. This approach can be used totrack such small objects so they can be classified rather than filteringthem out.

In reference to FIG. 4, with the sensor calibration result disclosedelsewhere herein, the foreground blobs can be transformed to 3D worldcoordinates, and their actual heights and volumes can be estimated (step56). Morphological filtering can be used to remove a blob if selectedcharacteristics, such as height, width, aspect ratio, volume,acceleration, velocity, and/or other spatiotemporal characteristics areoutside a detection threshold (e.g., dynamically calculated threshold,static threshold, or the like).

Geometric filtering can be applied to further remove spurious blobsoutside the scene boundary (step 58). The depth background defines a 3Dscene boundary of the environment. A blob representing a real objectshould be within the 3D boundary. That is, if a blob's depth is largerthan the depth of the corresponding location of the depth background,then the blob is outside of the 3D boundary and can be removed, e.g., ablob detected from reflective surfaces such as a mirror. Passengers orother moving objects can then be readily detected by a backgroundsubtraction technique with high robustness to illumination change,shadows, and occlusion, to thereby provide accurate passenger data. Tofurther increase detection robustness, temporal information canalternatively or additionally be utilized, e.g., by tracking

Passenger tracking may also be based on the binary foreground map and amethod such as a Kalman filter to track passengers and estimate thespeed and moving direction thereof. Based on detection, tracking, andcounting, passenger data such as the presence of a passenger in thelobby, an estimated time of arrival (ETA), and a number of waitingpassengers can be obtained. Such passenger data can then be used to, forexample, improve lobby call registration and elevator dispatching.

For example, the detection, tracking, and counting, facilitated by thedepth-sensing device may facilitate registering a hall call for anapproaching passenger, particularly at a terminal floor; opening the cardoors for an approaching passenger if a car is already at the floor;prepositioning a car based on an approaching passenger; and/orgenerating multiple hall calls based on the number of approachingpassengers such as when multiple passenger essentially simultaneouslyleave a seminar.

In another disclosed non-limiting embodiment, the sensor 62 can beinstalled with a FOV toward the elevator doors 24 and the lobby H. Sucha position facilitates continuous monitoring of the lobby H such thatinformation may be obtained far more completely and further in advanceof that which is available by a sensor in car which can sense the lobbyH only when elevator doors are open. Similar processes may alternativelyor additionally be employed as above, but specifically designed andtrained for the 3D depth map data.

With reference to FIG. 6, in another disclosed non-limiting embodiment,a sensor system 30B may include a passenger tracking system 70 withinthe elevator car 22 to facilitate operation of the elevator doors 24.The passenger tracking system 70 may include a sensor 72 thatcommunicates with a data capture module 74, and a data processing module76 that communicates with the data capture module 74 and a door controlmodule 78. The passenger tracking system 70 can be a portion of thecontrol system 32, a stand-alone unit, or other system such as acloud-based system in communication with the control system 32.

Passenger tracking system 70 can be specially designed to utilize depthmap data. Tracking may be regarded as a Bayesian Estimation problem,i.e., what is the probability of a particular system state given theprior system state, observations, and uncertainties. In such tracking,the system state may be the position of the tracked object, e.g,location and, possibly, velocity, acceleration, and other objectcharacteristics, e.g., target features as disclosed elsewhere herein.The uncertainties are considered to be noise. Depending on whatsimplifying assumptions are made for mathematical tractability orefficiency, the Bayesian Estimation becomes the variants of KalmanFiltering (assumption of Gaussian additive noise) or the variants ofParticle Filtering (assumption of non-Gaussian noise). In 2D and 3Dobject tracking, where there are many pixels/voxels on target, thesystem state often includes a target representation that includesdiscriminative information such as color descriptors (2D only), shapedescriptors, surface reflectivities, etc. The possible target models aresensor and application specific.

One disclosed non-limiting embodiment of depth data tracking forpassenger tracking system 70 is based on Kalman Filtering and the systemstate includes five (5) variables: x, y, h, vx and vy, which representtarget's real world x and y position, height, and velocities in the xand y directions. The tracking process comprises two steps: predictionand update. A constant velocity model, or other types of model such asrandom walk or constant acceleration models, can be applied forprediction and, through the model, targets (their states) in a previousdepth map can be transferred into the current depth map. A more complexmodel can be used if needed. In the update step, first all the targetsin the current depth map are detected with an object detection process,i.e., depth based background subtraction and foreground segmentation, asdisclosed elsewhere herein, then the detected targets are associatedwith predicted targets based on a global optimal assignment process,e.g. Munkres Assignment. The target's x, y, and h variables are used asfeatures for the assignment. The features (x, y, and h) are effective todistinguish different targets for track association.

For the predicted target that has an associated detected target, thetarget system state can be updated according to the Kalman equation withthe associated detected target as the observation. For a predictedtarget that has no associated detected target, the system state may staythe same, but the confidence of target will be reduced, e.g., for atarget that is already going out of the field of view. A track will beremoved if its confidence falls below a predetermined or selected value.For a detected target that has no associated predicted target, a newtracker will be initialized.

Other tracking approach like Particle Filtering may alternately oradditionally applied which will be more robust in cases where a targetabruptly changes its velocity. The Kalman approach, requires relativelylittle computational cost and may therefore be more suitable forreal-time application.

In this embodiment, the sensor 72 may be installed at the top of anelevator car 22 with a FOV downward and toward the elevator doors 24.The sensor 72 can thereby be operable to perceive passengers in the car22 and also, when the elevator doors 24 are open, may be operable toperceive passengers in the lobby H. The data capture module 74 capturesdata, e.g., 3D depth map data, from the sensor 72. When the door controlmodule 78 sends a signal to open the doors 24, for example afterelevator 22 stops at a floor, the door control module 78 may alsotrigger a signal for the data capture module 74 to capture sensor data.In one embodiment, passenger tracking may only be active when the doors24 are open and/or may be inactive when the doors 24 are closed. Inanother embodiment, the data capture module 74 may continuously processdata and thereby detect when the doors 24 are open, eliminating the needfor this information from the door control module 78, such that the doorcontrol module 78 is free of the door position information.

With reference to FIG. 7, in another disclosed non-limiting embodiment,process 80 for detecting objects in the elevator car 22 and in the lobbyH is disclosed in terms of functional block diagrams and it should beappreciated that these functions can be enacted in either dedicatedhardware circuitry or programmed software routines capable of executionin a microprocessor based electronics control embodiment

The data capture module 74 communicates the data to the data processingmodule 76 to detect objects in both in the elevator car 22 and in thelobby H (step 82). The object detection may include foregrounddetection, as disclosed elsewhere herein, and passenger detection usingcomputer vision processes for depth data. Passenger detection may beachieved by human model fitting, e.g., by using a Deformable Part Model,and classification, where the detection and classification can bespecially trained for the FOV and 3D depth map data.

Next, the detected objects will be tracked to obtain their moving speedand direction (step 84). The speed and direction can be in the sensorcoordinate system, and/or through sensor calibration, in the worldcoordinate system as further disclosed elsewhere herein. If detectedpassengers are just standing in the elevator car 22 or the lobby H,their moving speed is 0, which indicates that these passengers are notimmediately going to board or exit the elevator car 22.

For depth map based tracking, various processes can be used as disclosedelsewhere herein. Particular motion detection functions, for example,using Bayesian Estimation, determines if a passenger is just shiftingposition, or is intentionally moving toward the doors 24 from within thecar 22. This is particularly beneficial to specifically identify if apassenger at the rear of a crowded car 22 who wishes to exit.

With the information of moving speed and direction of passengers both inthe elevator car 22 and in the lobby H, the elevator doors 24 may berespectively controlled (step 86). For example, if numerous passengersare boarding or exiting, the elevator doors 24 can be controlled toremain open relatively longer than normal and then be closed promptlyafter all the passengers have boarded or exited. Conversely, if thereare no passengers waiting to board or exit, the elevator doors 24 can becontrolled to close relatively more quickly than normal to reducepassenger wait time and improve traffic efficiency.

With reference to FIG. 8, in another disclosed non-limiting embodiment,a sensor system 30C may include an unoccupied car determination system90 to facilitate determination as to whether the elevator car 22 isunoccupied, as an unoccupied elevator car 22 may be advantageously movedfrom five to ten times faster than an occupied elevator car 22 or movedin other ways not comfortable to passengers and/or within coderestrictions.

The unoccupied car determination system 90 may include a sensor 92 thatcommunicates with a data capture module 94, and a data processing module96 that communicates with the data capture module 94, and a car statusmodule 98. The unoccupied car determination system 90 can be a portionof the control system 32, a stand-alone unit, or other system such as acloud-based system in communication with the control system 32. Theunoccupied car determination system 90 may additionally include a loadsensor 100 in communication with the data capture module 94 and the dataprocessing module 96.

With reference to FIG. 9, in another disclosed non-limiting embodiment,process 110 for determining elevator car 22 is unoccupied is disclosedin terms of functional block diagrams and it should be appreciated thatthese functions can be enacted in either dedicated hardware circuitry orprogrammed software routines capable of execution in a microprocessorbased electronics control embodiment

The load sensor 100 can be operable to sense a current load weight ofthe elevator car 22, and may further determine if the sensed load weightis less than a preset threshold. The load sensor 100 may further triggera signal to the data capture module 94 to indicate that there is a highprobability (e.g., greater than 80%, or, 90%, or 95%) that the elevatorcar 22 is empty (step 111). Should the data capture module 94 receive anempty signal from the load sensor 100, the data capture module 94 willpass the current depth map sensor view (step 112) to the data processingmodule 96 for further confirmation that the car 22 is empty viaapplication of data capture processes (step 113). The load sensor 100,however, may be a relatively course sensor and may tend to drift inaccuracy over time. If the load sensor 100 is sufficiently inaccurate,it may be desirable that data capture module 94 run continuously ratherthan being triggered by load sensor 100.

Utilization of a 3D depth-sensing sensor as the sensor 92 facilitatesconfirmation of an empty car by in-car foreground detection or passengerdetection, with various analytics processes modified to operate with thedepth data as disclosed elsewhere herein. The 3D depth-sensing sensorcan facilitate accurate identification of passengers, heretoforeundetectable objects (e.g., such as briefcases, umbrellas, luggage andthe like) or a combination comprising at least one of the foregoing.Such identification can be accompanied by an audible notification, forexample, “PLEASE REMEMBER YOUR BELONGINGS.” It should be appreciatedthat other appropriate alerts may alternatively be provided.

An output of the data processing module 96 can include a signalindicating whether the car 22 is confirmed unoccupied (step 114). Withthis signal, elevator standby mode, unoccupied movement modes, and/ormulticar functions can be accurately applied (step 120).

A signal from the data processing module 96 may additionally oralternatively be an input to the load sensor 100 for re-calibration tomaintain the accuracy thereof (step 116). For example, upon confirmationof an empty car 22 via the sensor 92, the load sensor 100 can berecalibrated. In particular, if the car 22 is confirmed empty, thesensed load weight by the load sensor 100 may be set to zero, or, thedifference may be used to adjust the offset in the load sensingequation.

In another disclosed non-limiting embodiment, an unoccupied carmanagement system 120 may be utilized to facilitate operation ofelevator car calls, car dispatching, and car motion, which are managedbased on the determination of whether the elevator car 22 is unoccupied.More specifically, the unoccupied car management system 120 can beutilized to cancel all remaining car call(s) when the car 22 isunoccupied, balance the number of passengers between cars 22, directpassengers to specific cars 22, and/or change a motion profile toprovide an enhanced passenger experience, improved dispatching, and/orincreased throughput.

With reference to FIG. 10, in another disclosed non-limiting embodiment,a sensor system 30D may include an elevator monitoring system 130 tofacilitate detection of objects and/or a trapped passenger within theelevator car 22. The elevator monitoring system 130 may include a sensor132, such as a 3D depth-sensing sensor. Utilization of the 3Ddepth-sensing sensor readily overcomes restrictions inherent in 2Dimaging, such as illumination changes, and occlusion as disclosedelsewhere herein.

The sensor 132 communicates with a data capture module 134, and a dataprocessing module 136 that communicate with the data capture module 132and a rescue center module 138. The system 130 can be a portion of thecontrol system 32, a stand-alone unit, or other system such as acloud-based system in communication with the control system 32.

An elevator operation monitoring module 140 may also communicate withthe data capture module 134. The elevator operation monitoring module140 monitors the status of the elevator system 20 and if there is anymalfunction, the elevator operation monitoring module 140 may triggerthe sensor 132. The data capture module 134, when triggered, willcapture one or more depth maps from the sensor 132 for communication tothe data processing module 136. The data processing module 136 receivesthe 3D depth map data and may apply various analytics processes todetermine whether there are any passengers or objects in the elevatorcar 22 as disclosed elsewhere herein. It is also possible for datacapture module 134 to run continuously without trigger from elevatoroperation monitoring module 140.

In a malfunction such as a power outage, a battery backup 142 may beprovided for continued 3D sensing and processing. The continued 3Dsensing and processing may thus be performed in a manner to conservebattery life by judicious use under loss-of-power conditions.

With reference to FIG. 11, in another disclosed non-limiting embodiment,a process 150 for operation of the elevator monitoring system 130 isdisclosed in terms of functional block diagrams and it should beappreciated that these functions can be enacted in either dedicatedhardware circuitry or programmed software routines capable of executionin a microprocessor based electronics control embodiment.

The process 150 provides for initial data processing to extract aforeground region based on depth background subtraction (step 152). Adepth background model may be generated a priori and updated asrequired. Generation of the depth background model may be based on, forexample, a codebook process. The depth background subtraction with anactive 3D sensor is advantageously robust to illumination changesbecause the transmitted signal is used to determine depth.

Next, a foreground region is segmented based on the depth map andspatial information (step 154). In this step, regions corresponding todifferent passengers or other objects such as luggage can be segmentedfrom the background. Finally, each segmented region is checked with ahuman shape model to determine whether the depth data is of a human(step 156). In one example, the depth-based human shape model can be aDeformable Part Model to increase robustness to occlusion. The partbased model may also be trained for the depth data and sensor FOV toincrease accuracy. Multiple models may be created for differentpassenger poses, like standing, sitting, and lying down. The results arethen output to indicate, for example, a number of passengers or objects(step 158). The data processing module 136 thus not only outputsinformation as to whether there is a trapped passenger in the elevatorcar 22, but also the number of passengers that are trapped forcommunication to the rescue center module 138 to facilitate anappropriate rescue response.

With reference to FIG. 12, in another disclosed non-limiting embodiment,a sensor system 30E can include a special loading system 160 tofacilitate the detection of special loading conditions. Special loadingconditions, as defined herein, may include loading any object other thana human passenger and any loading that takes a relatively longer timethan normal, e.g., for wheelchairs, the elderly, passenger with largeluggage carriages, etc.

With the detection of special loading conditions, the special loadingsystem 160 improves passenger experience and traffic performance. Forexample, an elevator dispatching system of the elevator control 32 canassign an elevator car 22 with sufficient free space and the elevatordoor control 78 (FIG. 6) can hold the elevator doors 24 open longer toaccommodate slowly moving passengers or other special loading conditionssuch as large luggage (which might even take multiple trips in and outof the car 22 to load), service carts, or even an autonomous vehicle.

The special loading system 160 may include a sensor 162 (installed inthe lobby H or at a remote kiosk) to view a passenger who desires anelevator car 22 through analytics disclosed elsewhere herein.Utilization of a 3D depth-sensing sensor as the sensor 162 overcomes theaforementioned fundamental limitations of 2D imagers.

The sensor 162 communicates with a data capture module 164, thatcommunicates with a data processing module 166 that communicates withthe data capture module 164 and a special loading detection module 168.The special loading detection module 168 may also receive informationfrom a classifier module 170 and communicates with an elevator controlsystem 172. The system 160 may be a portion of the control system 32, astand-alone unit, or other system such as a cloud-based system incommunication with the control system 32.

With reference to FIG. 13, in another disclosed non-limiting embodiment,a process 180 for operation of the special loading system 160 isdisclosed in terms of functional block diagrams and it should beappreciated that these functions may be enacted in either dedicatedhardware circuitry or programmed software routines capable of executionin a microprocessor based electronics control embodiment.

Initially, in response to detection of a passengers desire to summon anelevator car as disclosed herein (step 182), the special loading process180 will acquire depth map data from the sensor 162 (step 184) and thenthe depth map data is communicated to the data processing module 166(step 186). The data processing module 166 then operates to segmentforeground objects from the background as disclosed elsewhere herein(step 168). This facilitates focus on foreground objects and removes thebackground influence. Various background modeling and subtractionprocesses suitably modified and trained on depth data can be applied tosegment foreground objects as disclosed elsewhere herein.

After foreground objects have been segmented, a spatial, orspatiotemporal classification approach facilitates detection of whetherthese foreground objects constitute a special loading condition (step190). For the general case of a special loading condition, it may bedifficult to manually define useful features for all possible specialloading conditions and to encompass the large amount of possiblevariation in the sensor data and environment. Therefore, the specialloading process 180 may be trained to learn features or featurehierarchies of special loading conditions that are different from normalloading.

With such automatically learned features, special loading detection maybe effectively classified by the classifier module 170 (step 192). Theclassification step 190 may be, for example, feature learning andclassification such as via a Deep Learning Network or Sparse LearnedDictionary. Other classifiers as known in the art may be advantageouslyemployed. The classifier training may, for example, be performed offlinefor various objects, and for real-time detection, the object detectioncan be specifically tailored based on predetermined requirements. Thispermits the special loading system 160 to be more adaptable for variousspecial loading detection needs as well as readily provide scalability.

Further, the detected special loading condition may be mapped to thefloor area adjacent to the elevator. Such map mapping may include, forexample, distances from a call button kiosk, and actual moving speed, sothat the elevator control system 172 may be tailored for the particulardispatching decisions and motion/door control (step 194). For example,this may be performed in one step. For example, the classifier, onrecognizing each special loading condition, directly outputs the learnedneeded floor area and actual moving speed. In an alternative embodiment,this may be performed in two steps, first the special loading conditionis classified then subsequent processing of the sensor data isconditioned on the special loading condition, to compute, for example,floor area, speed, or other information.

In one example, and with reference to FIG. 14, a special loadingcondition such as a passenger with a luggage cart “L” who presses abutton at a kiosk “K” may be tracked to obtain the moving speed “S,” tothereby provide an ETA (estimated time of arrival) from the distance “D”to the elevator car 22. The ETA can thus be used for appropriatedispatching and door control with adequate dwell time.

With reference to FIG. 15, in another disclosed non-limiting embodiment,a sensor system 30F may include an auto-calibration system 200 tofacilitate accurate determination of key calibration parameters ratherthan relying on an installer's labor, skill, and additional equipment.

The auto-calibration system 200 may include a sensor 202 such as a 3Ddepth-sensing sensor that can perform other functions such as thosedisclosed elsewhere herein. The sensor 202 may be deposed within anelevator car 22 or within an elevator lobby H. The sensor 202communicates with a data capture module 204, and data capture module 204communicates with a data processing module 206 and may communicate withan auto-calibration process 210. Data processing module 206 may alsocommunicate with auto-calibration process 210. The auto-calibrationsystem 200 can be a portion of the control system 32, a stand-aloneunit, or other system such as a cloud-based system in communication withthe control system 32.

The data processing module 206 may include a process 210 (FIG. 16) foroperation of the auto-calibration system 200. In another disclosednon-limiting embodiment, an process 210 for auto-calibration of sensor202 is disclosed in terms of functional block diagrams and it should beappreciated that these functions may be enacted in either dedicatedhardware circuitry or programmed software routines capable of executionin a microprocessor based electronics control embodiment

Initially, at least one measurement in the sensor coordinate system maybe determined by the system 200 of a moving object in the field of viewusing background subtraction and foreground segmentation as disclosedelsewhere herein. Next, data to establish a mathematical relationship,such as a transform matrix which captures the calibration information,is recorded in the sensor coordinate system (u,v,d) pertaining to themovement of passenger through the world coordinate (x,y,z) space (step214).

Next, suppositions about the scene geometry, e.g., the floor is flat, apassenger stands upright on the floor, a passenger does not changeheight, doors are orthogonal to floors, etc., are utilized forcomparison of the recorded sensor coordinate system data withstatistical data about passenger's heights (FIGS. 17 and 18; step 216).Upright passengers are detected by, e.g., connected componentssatisfying a simple aspect ratio threshold. Once enough uprightpassengers are detected, the floor plane can be determined and for eachfloor location the distribution of passenger's heights can be computed.

From predetermined knowledge of the distribution of passenger's heights(FIG. 18), the Z-axis can be calibrated (step 218). This Z-axiscalibration from the distribution of passenger's heights can beconsidered a system identification problem where the requisitepersistent and sufficient input is the size and motion of passengerthrough the field of view. The recorded height data can be collectedduring a setup period, maintained over a time period, and/or be subjectto a forgetting factor.

From the apparent height as a function of range, or voxel aspect ratio,the (X, Y) axes can then be calibrated based on the Z-axis calibration(step 220). The sensor coordinate data may then be mapped into the worldcoordinate system of absolute or ‘metric’ units (step 222).

To further facilitate recognition of passenger's intention such asapproaching, leaving, or passing by, the position of the elevator doors24 may also be determined. The position of the elevator doors 24 may bedetermined based on various methods, such as detecting the locationwhere passengers appear, disappear, depth change detection, depth of anelevator car, elevator door horizontal movement, and shape recognition.That is, the deduction of scene geometry may also be extended to locatedoors, the edge of the field of view, etc. Further, any of thesetechniques can be combined with installer input, where convenient. Themethod can monitor the convergence of the matrix mathematicalrelationship estimation of the calibration information to determine whensufficient accuracy has been achieved.

In an alternative embodiment, the floor plane and the elevator doorposition can be estimated in the sensor coordinate system (u,v,d) andall tracking can be performed in this coordinate system. In this casethe estimated arrival time can be learned by timing passenger's tracks,e.g., as a function of an empirical map.

In an alternative embodiment, the position of the elevator doors 24 canbe established at the time of commissioning by having the installerfollow a standard operating procedure whereby a calibration rig ispositioned with respect to the elevator door 24. For example, the rigcan be positioned flush with the center of the elevator doors 24 andoriented perpendicularly from the elevator doors 24. Additional featurescan be utilized to indicate each of the calibration points on thecalibration rig with uniquely identifiable features, such as the use ofcolors, shapes or patterns such as QR codes.

In another alternative embodiment, other areas of interest besides theelevator doors 24 can be identified. For instance, the location ofpassenger fixtures such as the COP 28, destination entry kiosks, thelocation of escalator entry/exit landings, the location ofturnstiles/access control devices, room entrances, doorways, etc. can bespecified.

With reference to FIG. 19, in another disclosed non-limiting embodiment,a sensor system 30G may include a passenger tracking system 230 todetect a passenger in the lobbies H and the elevator cars 22 to link allthe information together to generate a traffic list (FIG. 20) for eachindividual in a building for various applications. For example, trafficpattern prediction based on the traffic list information can focus onthe whole building level passengers' traffic information instead ofsingle zones or multiple zones. The traffic list information providesmore detailed information about passenger's behaviors in the building,and also can be used for various applications in addition to elevatorcontrol and dispatching.

The passenger tracking system 230 may include a plurality of sensors 242that communicate with the elevator system 20 via the control system 32.In one example, a sensor 242 is located in each lobby H and eachelevator car 22. Alternatively, a sensor 242 is only located in eachelevator car 22. The sensors 242 can be 2D imagers, 3D depth sensingsensors, or any combination thereof.

With reference to FIG. 21, in this disclosed non-limiting embodiment, aprocess 250 for operation of the passenger tracking system 230 isdisclosed in terms of functional block diagrams and it should beappreciated that these functions can be enacted in either dedicatedhardware circuitry or programmed software routines capable of executionin a microprocessor based electronics control embodiment.

A traffic list (FIG. 20) contains detailed information of eachindividual passenger that has used an elevator, such as arrival time,origin lobby, destination lobby, etc. To generate the traffic list, eachindividual passenger is tracked from an initial point in a lobby, towhen the passenger leaves a destination lobby, as well as through anin-car track between the origin lobby and the destination lobby.

To generate the tracking list, the sensors 242 may collect passengerinformation based on various passenger detection and tracking processesas disclosed elsewhere herein. Initially, each person can be detectedand tracked when they appear in a lobby or upon exit from an elevatorcar 22 (step 252). If sensor 242 is a 3D depth sensor, the detection andtracking process disclosed elsewhere herein can be applied. If sensor242 is a 2D imaging sensor, “integral channel features” may be computedby multiple registered sensor channels via linear and/or non-lineartransformations of input images, then a passenger detection model basedon the “integral channel features” can be learned by boosting, whichoffers a robust and fast approach for learning given a large number ofcandidate features, and results in fast detectors when coupled withcascade classifiers. This detection and tracking process may, forexample, be based on 2D RGB video.

In one embodiment, two trackers are designed to track one target: ahead-shoulder tracker via online boosting, and a body tracker based onparticle filtering. A spatial constraint may also be utilized to combinethe two trackers and a boosted online classifier may be maintained forocclusion and disappearance judgment.

For example, when a person enters an elevator car, in-car detection andtracking is triggered (step 254). That is, each person is tracked whilewithin the car, and while the person is in the destination lobby (step256). For the in-car track, the sensor is looking relatively downward,so passengers will look similar as only the head and shoulder appear inthe field of view. This may complicate tracking when passengers arecrowded therein. To address this complication for a 2D image sensor,each passenger's head is first detected by, for example, a circle-Houghtransform, then optical flow based motion estimation is developed tofilter out motionless candidates and adjust a head detection result toenclose each passenger. To further facilitate the in-car tracking, amotion-guided particle filtering approach may combine two features,e.g., an HSV color histogram and an edge orientation histogram, and mayutilize an effective model updating strategy based on motion estimation.

In order to maintain the association of a person tracked in one sensor'sFOV with the same person tracked in another sensor's FOV, lobby/hallwaytracking and in-car tracking are associated as a passenger moves from alobby/hallway into a car and vice versa. The 2D image sensor hand offassociation problem may utilize visual surveillance and techniques forboth overlapping and non-overlapping fields of view and for bothcalibrated and non-calibrated fields of view. In one example, adescriptor, e.g. a feature vector, may be computed using color or shapeand then this descriptor is used to compute the correct associationacross the different fields of view.

In 3D tracking, the common 2D descriptors such as color and 2D projectedshape (e.g., 2D gradients) are not available. As such, a 3D descriptor,i.e., a surface reflectivity histogram, a Histogram of Spatial Oriented3D Gradients (HoSG3D), etc. may be used. The HoSG3D is different thanthe 2D HoG3D descriptor because the 3rd dimension is spatial, while inHoG3D, the 3rd dimension is time. However, passenger shape passenger maybe sufficiently similar that using only HoSG3D may not be sufficientlydiscriminative to unambiguously hand a track from one sensor to another.

In another embodiment, the natural serialization of passengers enteringan elevator car may be used to associate tracks, e.g., the first losttrack in one sensed volume is associated with the first newly acquiredtrack in the other sensed volume, etc. This, too, may not besufficiently accurate since passengers might exchange order while out ofboth sensed volumes, and the strict serialization of car entry may notoccur. To ensure accuracy, overlapping, calibrated sensed volumesprovide better performance since the position of an object in theoverlapping sensed volumes can be known to be at the same spatialposition.

In another embodiment, a combination of the above techniques, or can beused. When the multiple techniques provide conflicting information onthe correct track association, the ambiguity may be resolved by solvinga Bayesian Estimation problem to maximize the probability of correctassociation given the observations and uncertainties. It will berecognized that other mathematical formulations of the associationproblem are possible. For tracking hand over between a sensor 242Alocated in a lobby and a sensor 242B located in an elevator car 22, agraph based optimization approach may be utilized (FIG. 22). The graphbased optimization approach, in one example, includes three layers ofnodes, representative of tracking in the origin lobby, tracking in-car,and tracking in a destination lobby.

The tracking hand over is then solved by a graph-based optimization 260to find overall best paths. The example graph-based optimization 260 maybe weighted by order and time difference. That is, as passengerstypically enter and leave the car in a sequential manner, filteringthereof is readily achieved to provide best paths by weights andsimilarity of nodes.

With reference to FIG. 23, if the elevator doors 24 are opening, thenthe vertical edges of door 24, e.g., as detected by a line-based HoughTransform, will traverse regions 1, 2 and 3 in order, and if the door isclosing, the door edges will traverse regions 3, 2, 1 in order. Theposition of the elevator doors 24 may also be confirmed via a sensor242B located in elevator car 22 or a sensor 242A located in lobby H witha view of elevator doors 24 to confirm the doors are opening, opened,closing, closed. That is, the elevator door status may be input fromelevator controller 32 or may be detected by sensor 242A/242B to improvethe performance and efficiency of a tracking hand over solution. Forexample, the tracking hand over need only be performed when the elevatordoor is open. It should be appreciated that other conveyances will alsobenefit herefrom.

With reference to FIG. 24, in another disclosed non-limiting embodiment,a sensor system 30H may include a fusion based passenger tracking system270 to predict the potential movement of a passenger, then adaptivelyassign elevator cars based on instantaneous needs so as to bring moreefficiency and convenience to elevator passengers in the building. Anelevator system with a complete, accurate traffic list (FIG. 20) canpredict the potential movement of passengers on, for example, an hourly,daily, weekly, etc. basis and use the elevators based on the anticipatedtraffic to increase efficiency and convenience to elevators passengers.In order to achieve the robust traffic list generation, a fusion basedtraffic list generation method is provided.

Referring now to FIG. 25, the fusion based passenger tracking system 270may include a plurality of security sensors 280 a-280 n that communicatewith the elevator system 20 via the control system 32. That is, thesensor data from the security sensors 280 essentially provides data tothe control system 32 to include, but not be limited to, facialrecognition, badge identification, fingerprints iris data, security cardinformation, etc. In areas without surveillance coverage or where theanalytics processes may not perform well, the additional securitysensors can recognize the person and then, using sensor fusion, closethe gaps in the traffic list to make the whole process more robust. Inany instance where identity is associated with a passenger, the identityand associated passenger tracking data is maintained in a way thatpreserves privacy by using encryption, authentication, and othersecurity measures.

The sensor fusion may be performed by Bayesian Inference, but inalternative embodiments may be performed by any well-known technique.With the security information and traffic history data, the patterns fora person moving in the building may be determined to understand normalbehavior as well as improve elevator service. In this disclosednon-limiting embodiment, the traffic list contains detailed informationof passengers using the elevator sensors 284, as well as security datafrom various security sensors 280. The data from various sensors arefused and communicated to the elevator system 20 via the control system32. The identification information is linked with this person's visualdescription features, so the whole traffic list under different imager'sor sensor's views will have the ID information. That is, the passengertraffic list is based on coordinating (“hand-over”) between lobby andcar tracking results. The fused data may then be used to facilitateelevator dispatching.

Hand-over rules may be pre-defined, such as a first-in and first-outrule. For a first-in and first-out rule, when the lobby sensor and thecar sensor operate simultaneously for target tracking in the sameregion, and one passenger is moving from the lobby to board the car,then this out-of-lobby-into-car information may be used to link thetracker from the lobby to the tracker in the car. When the passengerleaves the car and goes to the lobby, a similar rule(out-of-car-into-lobby) may be applied to link the tracker in the carwith the tracker in the lobby.

In one example, security sensors recognize a particular passenger andhis security data is shared with all the other sensors to link thetracking results with that passenger's ID. Second, in some areas, wherethe security sensors are not installed, the security credentialinformation may be utilized to continue tracking that passenger'spresence in the building and in this way continue the traffic listgeneration for that passenger. Additional information derived from oneimager's or sensor's view may also be shared with other imager(s) orsensor(s) to further improve track association across non-overlappingviews.

The traffic lists for a single passenger may be combined over time, withtime as a parameter, using Bayesian Inference for a probabilisticprediction of the passenger's intended destination. Such a system couldlearn that passenger A always goes to Floor N early in the morning,typically to Floor C (the cafeteria) at noon, and always goes to theParking Garage in the late afternoon.

Further, the traffic lists for multiple passengers can be combined overtime, with time as a parameter, again using Bayesian Inference. Such asystem facilitates statistical distribution determination of elevatorusage for the entire building during a typical day as well as weekenddays, holidays, etc. This information could be used to pre-assign carsto runs (even purposefully skipping floors), for efficient parking,dispatching cars, etc.

Given the information of the traffic lists, the elevator optimization isachieved by techniques for real-time solution of an optimizationproblem. The traffic lists information can also be utilized for otherelevator related applications such as elevator daily load estimation toprovide one accurate energy report for future energy saving, elevatorsystem diagnostics based on abnormal traffic list information,modernization value propositions, and so on.

With reference to FIG. 26, in another disclosed non-limiting embodiment,a process 300 may further utilize the elevator sensors 284, as well asthe security data from the various security sensors 280 to recognize aparticular passenger for passenger convenience, to optimize elevatoroperations, improve operations and/or for various security purposes. Theprocess 300 permits multiple passengers to be simultaneously allowed inthe car without confusion of destinations.

Initially, a passenger may be recognized in an origin lobby such aswhile approaching an elevator (step 302). The elevator sensors 284 canbe simultaneously operable, disparate-view, multi-sensor recognition,particularly combining 2D imagers; and 1D, 2D, or 3D depth sensors aswell as alternative or combinations thereof, i.e., 2D/3D. Again, thedata from various imagers and depth sensors are fused and communicatedto the elevator system 20 via the control system 32. The passenger maybe recognized by, for example, something they know, e.g., a password,something they have, e.g., a token or ID card, and/or by something theyare, e.g., a unique biometric. In one biometric example, facerecognition is both relatively inexpensive and well developed.

Next, a call for a predefined destination floor is registered based onthe recognized person (step 304). The determination of the desired floorcan be prerecorded by the person or can be automatically learned bytraffic analytics such as via a traffic list. Even with recognition andtracking capabilities, a pattern for the particular individual may notbe automatically discernable without statistical analysis capable ofignoring outliers, i.e., due to occasional non-typical elevator usage.In one embodiment, Robust Principle Components Analysis (RPCA) for thisoutlier-ignoring learning is utilized. In another embodiment BayesianInference can be utilized.

Next, a particular elevator car is assigned for the person, and theperson is directed to the appropriate car (step 306). Variousassignments can be based on particular identification, common usage,etc. such that a particular passenger is always directed to the closestcar, fastest car to his or her destination, etc.

The assigned car can be further provisioned with an alert if the personboards the wrong car or is heading towards the wrong car. The alert canbe based upon tracking the passenger into the car, however, the alertneed not be a request to exit as such a request may result in a negativecustomer experience. The alert, in one example, can be used toderegister the passenger in the previous car and register the intendeddestination floor in the new car 22. The elevator dispatching can thenbe re-optimized in real-time, including redirecting the passengerthrough sky lobbies, to provide a desired throughput and customerexperience.

Next, the passenger may be tracked from the lobby, into the car, duringtransit, then through the destination lobby as discussed above. In someinstances, while the passenger is tracked within the car, selection of adifferent destination can be identified. For example, while tracking thepassenger to correspond with the destination, analytics that the personhas pushed a button to change destination, and temporally correlatedinformation from the car controller as to which button was pressed canbe used to identify the change in destination. Once the change indestination is identified, throughput optimization can be performedthereby.

The process 300 may also alert a passenger if, for example, thepassenger mistakenly exits at a destination different than thatregistered for that particular passenger (step 308). In one embodiment,it can be desirable to alert the passenger before the passenger actuallymistakenly exits. The process 300 may thereby infer the intent orinitial movement of the passenger toward the doors via track analysis,e.g., movement towards the front of the car. The customer alert may beby audible voice signal. Alternatively, for security purposes, the alertmay silently notify security personnel or systems and track thepassenger.

The elements disclosed and depicted herein, including in flow charts andblock diagrams throughout the figures, imply logical boundaries betweenthe elements. However, according to software or hardware engineeringpractices, the depicted elements and the functions thereof may beimplemented on machines through computer executable media having aprocessor capable of executing program instructions stored thereon as amonolithic software structure, as standalone software modules, or asmodules that employ external routines, code, services, and so forth, orany combination of these, and all such implementations may be within thescope of the present disclosure.

It should be appreciated that relative positional terms such as“forward,” “aft,” “upper,” “lower,” “above,” “below,” “bottom”, “top”,and the like are with reference to the normal operational attitude andshould not be considered otherwise limiting.

It should be appreciated that like reference numerals identifycorresponding or similar elements throughout the several drawings. Itshould also be appreciated that although a particular componentarrangement is disclosed in the illustrated embodiment, otherarrangements will benefit herefrom.

Although the different non-limiting embodiments have specificillustrated components, the embodiments of this invention are notlimited to those particular combinations. It is possible to use some ofthe components or features from any of the non-limiting embodiments incombination with features or components from any of the othernon-limiting embodiments.

Although particular step sequences are shown, disclosed, and claimed, itshould be appreciated that steps may be performed in any order,separated or combined unless otherwise indicated and will still benefitfrom the present disclosure.

The foregoing description is exemplary rather than defined by thelimitations within. Various non-limiting embodiments are disclosedherein, however, one of ordinary skill in the art would recognize thatvarious modifications and variations in light of the above teachingswill fall within the scope of the appended claims. It is therefore to beappreciated that within the scope of the appended claims, the disclosuremay be practiced other than as specifically disclosed. For that reasonthe appended claims should be studied to determine true scope andcontent.

What is claimed is:
 1. A passenger conveyance system, comprising: adepth-sensing sensor for capturing 3D depth map data of objects within afield of view that comprises a passenger waiting area adjacent to apassenger conveyance cab door; a processing module in operablecommunication with the depth-sensing sensor to receive the 3D depth mapdata, the processing module using the 3D depth map data to track anobject, determine an object parameter of the tracked object, andcalculate passenger data associated with the tracked object; and apassenger conveyance controller in communication with the processingmodule to receive the passenger data, the passenger data comprises anestimated arrival time, and a number of passengers waiting for apassenger conveyance cab, wherein the passenger conveyance controllercontrols a passenger conveyance dispatch control function in response tothe passenger data to assign a passenger conveyance cab with appropriatefree space.
 2. The system as recited in claim 1, wherein thedepth-sensing sensor comprises at least one of a structured lightmeasurement, phase shift measurement, time of flight measurement, stereotriangulation device, sheet of light triangulation device, light fieldcameras, coded aperture cameras, computational imaging techniques,simultaneous localization and mapping (SLAM), imaging radar, imagingsonar, scanning LIDAR, flash LIDAR, Passive Infrared (PR) sensor, andsmall Focal Plane Array (FPA).
 3. The system as recited in claim 2,wherein the field-of-view is about 180 degrees.
 4. The system as recitedin claim 3, wherein the object parameter comprises at least one of anobject count, a location, a size, a direction, an acceleration, avelocity, and an object classification.
 5. The system as recited inclaim 4, wherein the processing module determines the passenger databased on the object parameter of the tracked object, wherein thepassenger data provided to the passenger conveyance controller comprisesat least one of a probability of arrival, and a covariance.
 6. Thesystem as recited in claim 5, wherein the object parameter of thetracked object comprises object classification and the processing modulecalculates the passenger data if the object classification comprises apassenger.
 7. The system as recited in claim 6, wherein the processingmodule divides the field of view of the depth sensing sensor into afirst region and a second region, wherein the second region is definedas an area adjacent the passenger conveyance cab door.
 8. The system asrecited in claim 7, wherein the passenger data comprises a number ofpassengers waiting for a passenger conveyance cab, and wherein theprocessing module increments the number of passengers waiting for apassenger conveyance cab based on a number of tracked objects that enterthe second region.
 9. The system as recited in claim 8, wherein thedepth-sensing sensor is located on a wall adjacent to the passengerconveyance cab door.
 10. The system as recited in claim 9, wherein thedepth-sensing sensor is located at about knee height.
 11. The system asrecited in claim 10, wherein the depth-sensing sensor is located at 150mm to 700 mm from an adjacent floor.
 12. The system as recited in claim11, wherein the field of view comprises a passenger waiting area,adjacent to the passenger conveyance door.
 13. The system as recited inclaim 1, wherein the field of view comprises the passenger conveyancedoor and an area at least partially surrounding the passenger conveyancedoor.
 14. A method of providing video aided data for use in passengerconveyance control, the method comprising: detecting an object locatedin a field of view of a depth-sensing sensor wherein the field of viewcomprises a passenger waiting area adjacent to a passenger conveyancecab door; tracking the object based on distance to the passengerconveyance door; calculating passenger data associated with the trackedobject, the passenger data comprises an estimated arrival time, and anumber of passengers waiting for a passenger conveyance; communicatingthe passenger data to a passenger conveyance controller; and controllinga passenger conveyance cab with the passenger conveyance controller inresponse to the passenger data to assign a passenger conveyance withappropriate free space.
 15. The method as recited in claim 14, whereincontrolling the passenger conveyance cab further comprises at least oneof opening the passenger conveyance door, moving the passengerconveyance cab, stopping the passenger conveyance cab, and redirectingthe passenger conveyance cab.
 16. The method as recited in claim 15,wherein controlling further comprises dispatching two or more passengerconveyance cabs substantially simultaneously in response to thepassenger data.
 17. The method as recited in claim 16, whereincalculating passenger data comprises at least one of calculating anobject parameter for the tracked object, wherein the object parameterincludes location, a size, a velocity, a direction, an acceleration, andan object classification.
 18. The method as recited in claim 15, whereincalculating passenger data comprises background subtraction.
 19. Themethod as recited in claim 15, wherein calculating passenger datacomprises frame differencing.
 20. The method as recited in claim 15,wherein calculating passenger data comprises spurious data rejection.21. The method as recited in claim 20, wherein spurious data rejectionincludes: computing a depth background; segmenting a foreground object;removing a foreground region; segmenting a moving object by a 3Dmorphological operation; transforming the moving object to 3D worldcoordinates; estimating an actual height and actual volume of the movingobject; and removing a spurious moving object from a scene boundary bygeometric filtering.
 22. The method as recited in claim 21, wherein the3D morphological operation comprises computing a 2D foreground object bydepth background subtraction, size filtering on a mask as a function ofrange; connecting mask regions; segmenting objects in 3D based on depthdiscontinuity, or a combination comprising at least one of theforegoing.
 23. The method as recited in claim 22, wherein the 2Dforeground object within the mask is at any depth.